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Foreword 


ISO (the International Organization for Standardization) and IEC (the International Electrotechnical 
Commission) form the specialized system for worldwide standardization. National bodies that are 
members of ISO or IEC participate in the development of International Standards through technical 
committees established by the respective organization to deal with particular fields of technical 
activity, ISO and IEC technical committees collaborate in fields of mutual interest. Other international 
organizations, governmental and non-governmental, in liaison with ISO and IEC, also take part in the 
work. 


The procedures used to develop this document and those intended for its further maintenance 
are described in the ISO/IEC Directives, Part 1. In particular, the different approval criteria 
needed for the different types of document should be noted. This document was drafted in 
accordance with the editorial rules of the ISO/IEC Directives, Part 2 (see www.iso.org/directives or 
www.iec.ch/members experts/refdocs), 


Attention is drawn to the possibility that some of the elements of this document may be the subject 
of patent rights. ISO and IEC shall not be held responsible for identifying any or all such patent 
rights. Details of any patent rights identified during the development of the document will be in the 
Introduction and/or on the ISO list of patent declarations received (see www.iso.org/patents) or the IEC 
list of patent declarations received (see https://patents.iec.ch). 


Any trade name used in this document is information given for the convenience of users and does not 
constitute an endorsement. 


For an explanation of the voluntary nature of standards, the meaning of ISO specific terms and 
expressions related to conformity assessment, as well as information about ISO's adherence to 
the World Trade Organization (WTO) principles in the Technical Barriers to Trade (TBT) see 


www.iso.org/iso/foreword.html. In the IEC, see www.iec.ch/understanding-standards. 


This document was prepared by Joint Technical Committee ISO/IEC JTC 1, Information technology, 
Subcommittee SC 29, Coding of audio, picture, multimedia and hypermedia information. 


A list of all parts in the ISO/IEC 18181 series can be found on the ISO and IEC websites. 


Any feedback or questions on this document should be directed to the user's national standards 
body. A complete listing of these bodies can be found at www.iso.org/members.html and 
www.iec.ch/national-committees. 
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Introduction 


The ISO/IEC 18181 series, also known as "JPEG XL’, supports lossless and lossy compression of images 
and image sequences. 


This document provides the framework, concepts and methodology for testing codestreams and 
implementations, and the criteria to be achieved to claim conformance to the ISO/IEC 18181 series. 
The objective of this standard is to promote interoperability between JPEG XL decoders, and to test 
these systems for conformance. Conformance testing is the testing of a candidate implementation for 
the existence of specific characteristics required by a standard. 


The purpose of this document is to define a common test methodology, to provide a framework for 
specific test suites, and to define the procedures to be followed during conformance testing. 
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Information technology — JPEG XL Image Coding System — 


Part 3: 
Conformance testing 


1 Scope 

This document specifies the conformance testing of the ISO/IEC 18181 series, also known as JPEG XL. 
Other desirable aspects of implementation (including robustness and performance) are outside the 
scope of this document. 

2 Normative references 

The following documents are referred to in the text in such a way that some or all of their content 
constitutes requirements of this document. For dated references, only the edition cited applies. For 


undated references, the latest edition of the referenced document (including any amendments) applies. 


ISO/IEC 18181-1, Information technology — JPEG XL image coding system — Part 1: Core coding system 


3 Terms and definitions 

For the purposes of this document, the terms and definitions given in ISO/IEC 18181-1 apply. 

ISO and IEC maintain terminology databases for use in standardization at the following addresses: 
— ISO Online browsing platform: available at https://www.iso.org/obp 

— IEC Electropedia: available at https://www.electropedia.org/ 


4 Testing procedure 
4.1 Structure of the test corpora 


4.1.1 General 


NOTE JPEG XL specifies infinite-precision arithmetic, whereas practical implementations are likely to 
use finite-precision arithmetic, Allowing each intermediate operation to round the result to a floating-point 
representation might not give sufficient freedom for alternate implementation strategies, or conversely too much 
freedom to be useful. This document therefore specifies tolerances for the decoded samples. 


A description of the test corpora is provided in Annex A and the following files are available from 


— atestcases/ subdirectory, containing 
— multiple subdirectories, each of which contains a single test case for conformance testing 
— multiple txt files, each line of which contains the name of a folder corresponding to a test case 


— aconformance.py file that provides a possible implementation of the testing procedure described in 
this document 


— alcms2.py file, which is a supporting file for conformance.py 
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Each subdirectory contains: 
— exactly one JSON filel2], named test.json 

— exactly one JXL file, named input.jxl 

— up to two NPY files, named reference image.npy or reference preview.npy 

— upto two ICC profile files, named reference.icc or original.icc 

— zero or one JPEG files, named reconstructed.jpeg 

Ifa NPY file is present, then the ICC profile named reference.icc is also present. 
Collectively, the NPY, ICC and JPEG files are known as the reference for this test case. 
The JSON file specifies the tests to be performed as part of this test case. 


The NPY files contain sample values for either the main frames or the preview frames, represented 
as a four-dimensional array with dimensions (num, frames, height, width, num channels) respectively. 
num frames is always 1 for the preview frames. The data format stored in these arrays is one of float (32 
or 64 bits), uint8 and uint16. 


“reference. icc” represents the colour space of the data in the NPY files. 


“original.ice” represents the ICC profile of the samples that were used to create the JPEG XL file; this 
is metadata that may be present in the JPEG XL file. 


“reconstructed. jpg" contains the original JPEG1 file that produced a recompressed JPEG1 (i.e. with a 
"jbrd" box) JPEG XL file. 


4.1.2 Numpy File Format 


A NPY file begins with the sequence of bytes “93 4e 55 4d 50 59" (byte 93 followed by “NUMPY” in 
ASCII), followed by the sequence of bytes “01 00”, followed by two bytes representing a 16-bit little- 
endian integer LEN that represents the length of the rest of the header. 


The rest of the header contains the newline-terminated ASCII string “('descr': '<f4', *fortran 
order': False, ‘shape’: (X, Y, 2, W), )",where X, Y, Z, W represent integers in their usual base-10 
textual representation. 


The NPY file represents an array with dimensions (X, Y, Z, W). The rest of the file contains X x Yx Z x 
W little endian, 32-bit floating point numbers (as specified in IEEE 754-2019), in raster order with the 
W dimension varying faster, i.e. floating point number in position (x, y, z, w) is stored in the 4 bytes 
starting at position 4-(x-Y'Z-W + y-Z-W + z-W + w) after the header. 


NOTE This is a subset of the format specified in 


4.2 Image similarity 

Samples are compared in the colour space specified by the reference ICC profile “reference. icc”, with 
nominal values in the interval [0, 1] (no clipping is to be applied to values outside this range). An image 
is similar to the reference image if and only if the following three conditions are met: 

1) The image dimensions, number of frames and number of channels are identical. 


2) Each of their samples are similar, i.e. the peak error is bounded. A decoded sample D and reference 
sample R are similar if and only if |D - R| is smaller or equal to a given threshold. 


3) For all channels (components), the root mean square error (the root of the sum of (D - RJ? over all 
samples) is smaller or equal to a given threshold. 
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For the purposes of this comparison, the channels are assumed to be ordered as follows: the first 
three channels are the RGB channels (in the order R, G, B); then, channel 3+i is the i-th extra channel 
(ISO/IEC 18181-1:2022, C.4.10). 


In the case of greyscale images, the reference is given in RGB, where the RGB channels are equal (R = G 
=B). 


4.3 Conformance on a single test case 
The JSON file for a given test case represents the configuration of the given test case. 


The JSON file represents this configuration using a key-value format, where keys are strings and values 
are strings, numbers, arrays or nested key-value pairs. 


If the “frames” key is present, its corresponding value is an array of per-frame information. Its length 
equals the number of frames in “reference_image.npy". For each array entry, conformance is tested 
by checking similarity of the i-th decoded frame with the i-th frame in "reference image.npy", For the 
purposes of this document, the i-th decoded frame is defined to be the i-th frame with a FrameType 
of either “kRegular” or “kSkipProgressive" and either a non-zero duration or is last set to true, after 
any blending and orientation is applied (ISO/IEC 18181-1:2022, C.1, C.2). This corresponds to frames 
that are meant to be displayed to the end user. If the total number of decoded frames is not equal to 
the length of the array, the decoder is non-conforming. The keys in the i-th entry of the array influence 
conformance testing at follows: 


— The key “rms_error” specifies the threshold for the root mean square error of any channel. 
— The key “peak error" specifies the threshold for the peak error of any channel. 


— The value of the key “duration”, if present, specifies the amount of time (in seconds) that the frame 
should be displayed for (ISO/IEC 18181-1:2022, C.2). The decoder is not conformant if this duration 
differs from the duration reported by the decoder by more than 0.0001 seconds. 


— The value of the key “name”, if present, is a string that specifies the name of the frame 
(ISO/IEC 18181-1:2022, C.2). If it differs in either content or length from the decoded value, the 
decoder is not conformant. 


— The value of the key “timecode”, if present, is an integer that specifies the timecode of the frame 
(ISO/IEC 18181-1:2022, C.2). If it differs from the decoded value, the decoder is not conformant. 


If the "preview" key is present, its corresponding value is a single instance of the per-frame information 
that is used for the "frames" key. It is to be interpreted in the same way, except that it refers to the 
"reference preview.npy" file and the decoded preview image. 


Ifthe "original ice" key is present, its value is ignored. The decoder is not conformant if it reports that 
the original ICC profile of the samples represented by the JPEG XL file (ISO/IEC 18181-1:2022, Annex B) 
differs from “original . icc”, either in length or in contents. 


If the “reconstructed jpeg" key is present, its value is ignored. The decoder is not conformant if the 
process of reconstructing an ISO/IEC 18477-1 JPEG bitstream (as described in ISO/IEC 18181-2) differs 
from “reconstructed. jpg”, either in length or in contents. 


If the “intensity_target” key is present, its value is a number that defines the intensity target of the 
decoded image (ISO/IEC 18181-1:2022, Table A.18). If the decoded value differs by more than 0.0001, 
the decoder is not conformant. The “nin nits", “relative to max display" and "linear below" keys 
are defined in a similar way. 


If the "extra channel type" key is present, its value is an array of strings that has a length of “num_ 
channels-3" (i.e. equal to the number of extra channels). The i-th entry of the array corresponds to the 
“type” value of the ExtraChannellnfo (ISO/IEC 18181-1:2022, A.9) bundle corresponding to that extra 
channel. If any of those values differs from the decoded ones, the decoder is not conformant. 
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If the "bits per channel" key is present, its value is an array of integers representing the original bit 
depth. Its length is “num channels-2"; the first value corresponds to the bit depth.bits per sample 
value of the color channels (ISO/IEC 18181-1:2022, Table A.16), and the other values correspond to the 
bit depth.bits per sample value of the extra channels (ISO/IEC 18181-1:2022, Table A.22). 


If the "exp bits per channel" key is present, its value is an array of integers. Its length is “num 
channels-2"; the first value corresponds to the bit depth.exp bits value of the color channels 
(ISO/IEC 18181-1:2022, Table A.16), and the other values correspond to the bit depth.exp bits value 
of the extra channels (ISO/IEC 18181-1:2022, Table A.22). 


4.4 Encoder test 


As specified in ISO/IEC 18181-1, any encoding process is acceptable so long as it produces a valid 
codestream. Thus, an encoder shall be considered conforming if it produces output files which 
are successfully decoded by a conforming decoder. More precisely, the steps for testing encoder 
conformance are as follows: 


a) Select a test image that represents the type of imagery that the encoder is designed to compress. 
The reference decoded images provided for decoder conformance tests are acceptable but not 
required. 


b) Encode with the encoder under test. 
c) Send the codestream to the reference decoder. 
d) An encoder is found to be compliant if the reference decoder can fully decode the image. 


e) Repeat steps a) through d) for all parameters for which the encoder is designed. These parameters 
should be varied to the extent to which the encoder will be used. 


f) Repeat steps a) through e) for several test images, sampling the breadth of imagery types (small 
image size, large image size, odd image sizes, number of components, component bit depths, 
component sampling) the encoder is designed to compress. 


4.5 Decoder 


A decoder shall be considered conforming to Level 5 of the Main profile if it is conforming to all the 
testcases specified in the main level5.txt file available from -i 


A decoder shall be considered conforming to Level 10 of the Main profile if it is conforming to al the 
testcases specified in the main_leve110.txt file available from 1 5 
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Annex A 
(informative) 


Description of the test corpora 


Table A.1 describes what combinations of coding tools and peak/mean error values are found in each 
of the test corpora. Coding tools and values are considered on a per-channel basis; for example, in the 
Level 5 test corpus it is possible to find an image with the three colour channel encoded with VarDCT 
and a peak/average error bound of 0.06 and 0.02, plus one extra channel encoded with 8-bit Modular 
(with no filters) and a peak/average error bound of 1/219. 


In this table, “no filters" means that all the coding tools from ISO/IEC 18181-1:2022, Annexes J and K, as 
well as L.7 and L.8 are not enabled for that channel. In contrast, “with filters" means that some of those 
tools are enabled. An X is present in the “Level 5"/"Level 10" columns if images with this configuration 
can be found in the corresponding corpora. 


Table A.1 — Summary of parameters in test corpora 


Level5  [Level10 _ [Peak error RMSE 
VarDCT, Modular with filters [X X 0.06 0.02 
VarDCT, Modular no filters X x 0.06 0.02 
VarDCT, Modular with filters x 0.004 1/10* 
VarDCT with no filters X 0.004 1/105 
Modular 8-bit, no filters, RGB |X x 1/210 1/210 
Modular 10-bit, no filters, RGB |X X 1/22 1/22 
Modular 12-bit, no filters, RGB X x 1/24 1/24 
Modular 16-bit, no filters, RGB X 1/28 1/28 


Table A.2 lists the test cases and the relevant coding tools that are tested with them. The main levels. 
txt file contains all the test cases that are indicated to be level 5 in the rightmost column; the main_ 
levell0.txt file contains all the test cases. 


Table A.2 — Summary of test cases and relevant coding tools 


test case what it tests level 
alpha nonpremultiplied Modular mode, alpha channel, 12-bit 5 
alpha_premultiplied VarDCT mode, premultiplied alpha channel, 12-bit color, 10 
16-bit alpha 
alpha_triangles Modular mode, alpha channel, 9-bit 5 
animation_icos4D VarDCT mode, alpha channel, animation 5 
animation newtons cradle Modular mode, Palette, animation 5 
animation_spline Splines, animation 5 
bench_oriented_brg Container, VarDCT mode, JPEG reconstruction, Orienta-|5 
tion, ICC profile 
bicycles Modular mode, Squeeze, XYB 5 
bike VarDCT mode 5 
blendmodes Modular mode, various blend modes, 12-bit 5 
cafe Container, VarDCT mode, JPEG reconstruction, chroma] 5 
upsampling 
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Table A.2 (continued) 
cmyk layers Modular mode, CMYK (kBlack channel), alpha channel, 10 
layers, large ICC profile 
delta palette Modular mode, Palette with delta entries 5 
grayscale VarDCT mode, Grayscale ICC profile 5 
grayscale jpeg Container, VarDCT mode, JPEG reconstruction, Grayscale|5 
[grayscale public university Modular mode, Squeeze, Grayscale 5 
noise Noise 5 
opsin inverse OpsinInverseMatrix 5 
lossless pfm Modular mode, lossless 32-bit float 10 
1277 Flower Modular mode, 1277 5 
patches Container, VarDCT mode, Patches 5 
patches_lossless Container, Modular mode, Patches 5 
progressive VarDCT mode, LF frame (Modular + Squeeze), TOC per-|5 
mutation, HF passes 
sunset logo Modular mode, RCT, 10-bit, Orientation, two layers (alpha |5 
blending) 
spot Modular mode, two spot colors, two layers, 16-bit ProPhoto | 10 
upsampling Modular mode, Palette with delta entries 5 
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